<!DOCTYPE html>


<html lang="zh-CN">


<head>
  <meta charset="utf-8" />
    
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
  <title>
    使用Kubeadm设置Kubernetes集群的高可用性.md |  
  </title>
  <meta name="generator" content="hexo-theme-ayer">
  
  <link rel="shortcut icon" href="/favicon.ico" />
  
  
<link rel="stylesheet" href="/dist/main.css">

  
<link rel="stylesheet" href="https://cdn.jsdelivr.net/gh/Shen-Yu/cdn/css/remixicon.min.css">

  
<link rel="stylesheet" href="/css/custom.css">

  
  
<script src="https://cdn.jsdelivr.net/npm/pace-js@1.0.2/pace.min.js"></script>

  
  

  

</head>

</html>

<body>
  <div id="app">
    
      
    <main class="content on">
      <section class="outer">
  <article
  id="post-k8s/使用Kubeadm设置Kubernetes集群的高可用性"
  class="article article-type-post"
  itemscope
  itemprop="blogPost"
  data-scroll-reveal
>
  <div class="article-inner">
    
    <header class="article-header">
       
<h1 class="article-title sea-center" style="border-left:0" itemprop="name">
  使用Kubeadm设置Kubernetes集群的高可用性.md
</h1>
 

    </header>
     
    <div class="article-meta">
      <a href="/2020/11/11/k8s/%E4%BD%BF%E7%94%A8Kubeadm%E8%AE%BE%E7%BD%AEKubernetes%E9%9B%86%E7%BE%A4%E7%9A%84%E9%AB%98%E5%8F%AF%E7%94%A8%E6%80%A7/" class="article-date">
  <time datetime="2020-11-10T16:00:00.000Z" itemprop="datePublished">2020-11-11</time>
</a> 
  <div class="article-category">
    <a class="article-category-link" href="/categories/k8s/">k8s</a>
  </div>
  
<div class="word_count">
    <span class="post-time">
        <span class="post-meta-item-icon">
            <i class="ri-quill-pen-line"></i>
            <span class="post-meta-item-text"> 字数统计:</span>
            <span class="post-count">3.6k</span>
        </span>
    </span>

    <span class="post-time">
        &nbsp; | &nbsp;
        <span class="post-meta-item-icon">
            <i class="ri-book-open-line"></i>
            <span class="post-meta-item-text"> 阅读时长≈</span>
            <span class="post-count">17 分钟</span>
        </span>
    </span>
</div>
 
    </div>
      
    <div class="tocbot"></div>




  
    <div class="article-entry" itemprop="articleBody">
       
  <h1 id="使用Kubeadm设置Kubernetes集群的高可用性"><a href="#使用Kubeadm设置Kubernetes集群的高可用性" class="headerlink" title="使用Kubeadm设置Kubernetes集群的高可用性"></a>使用Kubeadm设置Kubernetes集群的高可用性</h1><p>当我们为生产环境在本地设置Kubernetes（k8s）集群时，建议以高可用性部署它。高可用性意味着在HA中安装Kubernetes主节点或工作节点。在本文中，我将演示如何使用kubeadm实用程序设置高可用性的Kubernetes集群。</p>
<p><strong>为了进行演示，我使用了五个具有以下详细信息的CentOS 7系统：</strong></p>
<ul>
<li>k8s-master-1 –最低CentOS 7 – 192.168.1.40 – 2GB RAM，2vCPU，40 GB磁盘</li>
<li>k8s-master-2 –最低CentOS 7 – 192.168.1.41 – 2GB RAM，2vCPU，40 GB磁盘</li>
<li>k8s-master-3 –最低CentOS 7 – 192.168.1.42 – 2GB RAM，2vCPU，40 GB磁盘</li>
<li>k8s-worker-1 –最低CentOS 7 – 192.168.1.43 – 2GB RAM，2vCPU，40 GB磁盘</li>
<li>k8s-worker-2 –最低CentOS 7 – 192.168.1.44 – 2GB RAM，2vCPU，40 GB磁盘</li>
</ul>
<p><img src="http://iubest.gitee.io/pic/640-1601017115245.png" alt="img"></p>
<p>注意：etcd集群也可以在主节点之外形成，但是为此我们需要额外的硬件，因此我将etcd安装在主节点内。</p>
<p>设置K8s集群高可用的最低要求：</p>
<ul>
<li>在所有主节点和工作节点上安装Kubeadm，kubelet和kubectl</li>
<li>主节点和工作节点之间的网络连接</li>
<li>所有节点上的Internet连接</li>
<li>根凭证或所有节点上的sudo特权用户</li>
</ul>
<p>让我们跳过安装和配置步骤。</p>
<h4 id="步骤1-设置主机名并在-etc-hosts文件中添加条目"><a href="#步骤1-设置主机名并在-etc-hosts文件中添加条目" class="headerlink" title="步骤1. 设置主机名并在/etc/hosts文件中添加条目"></a>步骤1. 设置主机名并在/etc/hosts文件中添加条目</h4><p>运行<code>hostnamectl</code>命令在每个节点上设置主机名，例如，示例显示K8s master节点:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ hostnamectl set-hostname &quot;k8s-master-1&quot;</span><br><span class="line">$ exec bash</span><br></pre></td></tr></table></figure>

<p>同样，在其余节点上运行上述命令并设置它们各自的主机名。在所有主节点和工作节点上都设置了主机名之后，请在所有节点上的<code>/etc/hosts</code>文件中添加以下条目。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">192.168.1.40   k8s-master-1</span><br><span class="line">192.168.1.41   k8s-master-2</span><br><span class="line">192.168.1.42   k8s-master-3</span><br><span class="line">192.168.1.43   k8s-worker-1</span><br><span class="line">192.168.1.44   k8s-worker-2</span><br><span class="line">192.168.1.45   vip-k8s-master</span><br></pre></td></tr></table></figure>

<p>我在主机文件中使用了另一个条目“ 192.168.1.45 vip-k8s-master”，因为我将在配置haproxy并在所有主节点上保持连接状态时使用此IP和主机名。该IP将用作kube-apiserver 负载均衡器ip。所有的kube-apiserver请求都将到达此IP，然后请求将分发到后端实际的kube-apiserver节点上。</p>
<h4 id="步骤2-在所有主-工作节点上安装和配置Keepalive和HAProxy"><a href="#步骤2-在所有主-工作节点上安装和配置Keepalive和HAProxy" class="headerlink" title="步骤2.在所有主/工作节点上安装和配置Keepalive和HAProxy"></a>步骤2.在所有主/工作节点上安装和配置Keepalive和HAProxy</h4><p>使用以下<code>yum</code>命令在每个主节点上安装keepalived和haproxy:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ sudo yum install haproxy keepalived -y</span><br></pre></td></tr></table></figure>

<p>首先在k8s-master-1上配置Keepalived，创建<code>check_apiserver.sh</code>脚本将获得以下内容：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@k8s-master-1 ~]$ sudo vi &#x2F;etc&#x2F;keepalived&#x2F;check_apiserver.sh</span><br><span class="line">#!&#x2F;bin&#x2F;sh</span><br><span class="line">APISERVER_VIP&#x3D;192.168.1.45</span><br><span class="line">APISERVER_DEST_PORT&#x3D;6443</span><br><span class="line"></span><br><span class="line">errorExit() &#123;</span><br><span class="line">    echo &quot;*** $*&quot; 1&gt;&amp;2</span><br><span class="line">    exit 1</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">curl --silent --max-time 2 --insecure https:&#x2F;&#x2F;localhost:$&#123;APISERVER_DEST_PORT&#125;&#x2F; -o &#x2F;dev&#x2F;null || errorExit &quot;Error GET https:&#x2F;&#x2F;localhost:$&#123;APISERVER_DEST_PORT&#125;&#x2F;&quot;</span><br><span class="line">if ip addr | grep -q $&#123;APISERVER_VIP&#125;; then</span><br><span class="line">    curl --silent --max-time 2 --insecure https:&#x2F;&#x2F;$&#123;APISERVER_VIP&#125;:$&#123;APISERVER_DEST_PORT&#125;&#x2F; -o &#x2F;dev&#x2F;null || errorExit &quot;Error GET https:&#x2F;&#x2F;$&#123;APISERVER_VIP&#125;:$&#123;APISERVER_DEST_PORT&#125;&#x2F;&quot;</span><br><span class="line">fi</span><br></pre></td></tr></table></figure>

<p>保存并退出文件，设置可执行权限：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ sudo chmod + x &#x2F;etc&#x2F;keepalived&#x2F;check_apiserver.sh</span><br></pre></td></tr></table></figure>

<p>备份<code>keepalived.conf</code>文件，然后清空该文件。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@k8s-master-1 ~]$ sudo cp &#x2F;etc&#x2F;keepalived&#x2F;keepalived.conf &#x2F;etc&#x2F;keepalived&#x2F;keepalived.conf-org</span><br><span class="line">[kadmin@k8s-master-1 ~]$ sudo sh -c &#39;&gt; &#x2F;etc&#x2F;keepalived&#x2F;keepalived.conf&#39;</span><br></pre></td></tr></table></figure>

<p>现在将以下内容粘贴到<code>/etc/keepalived/keepalived.conf</code>文件中</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@k8s-master-1 ~]$ sudo vi &#x2F;etc&#x2F;keepalived&#x2F;keepalived.conf</span><br><span class="line">! &#x2F;etc&#x2F;keepalived&#x2F;keepalived.conf</span><br><span class="line">! Configuration File for keepalived</span><br><span class="line">global_defs &#123;</span><br><span class="line">    router_id LVS_DEVEL</span><br><span class="line">&#125;</span><br><span class="line">vrrp_script check_apiserver &#123;</span><br><span class="line">  script &quot;&#x2F;etc&#x2F;keepalived&#x2F;check_apiserver.sh&quot;</span><br><span class="line">  interval 3</span><br><span class="line">  weight -2</span><br><span class="line">  fall 10</span><br><span class="line">  rise 2</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">vrrp_instance VI_1 &#123;</span><br><span class="line">    state MASTER</span><br><span class="line">    interface enp0s3</span><br><span class="line">    virtual_router_id 151</span><br><span class="line">    priority 255</span><br><span class="line">    authentication &#123;</span><br><span class="line">        auth_type PASS</span><br><span class="line">        auth_pass P@##D321!</span><br><span class="line">    &#125;</span><br><span class="line">    virtual_ipaddress &#123;</span><br><span class="line">        192.168.1.45&#x2F;24</span><br><span class="line">    &#125;</span><br><span class="line">    track_script &#123;</span><br><span class="line">        check_apiserver</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure>

<p>保存并关闭文件。</p>
<p>注意：对于master-2和3节点，仅需要更改此文件的两个参数。主节点2和3的状态将变为<code>SLAVE</code>，优先级分别为254和253。</p>
<p>在k8s-master-1节点上配置HAProxy，编辑其配置文件并添加以下内容：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@k8s-master-1 ~]$ sudo cp &#x2F;etc&#x2F;haproxy&#x2F;haproxy.cfg &#x2F;etc&#x2F;haproxy&#x2F;haproxy.cfg-org</span><br></pre></td></tr></table></figure>

<p>删除默认部分之后的所有行，并添加以下行</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@k8s-master-1 ~]$ sudo vi &#x2F;etc&#x2F;haproxy&#x2F;haproxy.cfg</span><br><span class="line">#---------------------------------------------------------------------</span><br><span class="line"># apiserver frontend which proxys to the masters</span><br><span class="line">#---------------------------------------------------------------------</span><br><span class="line">frontend apiserver</span><br><span class="line">    bind *:8443</span><br><span class="line">    mode tcp</span><br><span class="line">    option tcplog</span><br><span class="line">    default_backend apiserver</span><br><span class="line">#---------------------------------------------------------------------</span><br><span class="line"># round robin balancing for apiserver</span><br><span class="line">#---------------------------------------------------------------------</span><br><span class="line">backend apiserver</span><br><span class="line">    option httpchk GET &#x2F;healthz</span><br><span class="line">    http-check expect status 200</span><br><span class="line">    mode tcp</span><br><span class="line">    option ssl-hello-chk</span><br><span class="line">    balance     roundrobin</span><br><span class="line">        server k8s-master-1 192.168.1.40:6443 check</span><br><span class="line">        server k8s-master-2 192.168.1.41:6443 check</span><br><span class="line">        server k8s-master-3 192.168.1.42:6443 check</span><br></pre></td></tr></table></figure>

<p>保存并退出文件。</p>
<p><img src="http://iubest.gitee.io/pic/640-1601017115261.png" alt="img"></p>
<p>现在将这些三个文件（check_apiserver.sh，keepalived.conf和haproxy.cfg）从k8s-master-1复制到k8s-master-2和3</p>
<p>运行以下for循环将这些文件scp复制到master 2和3上；</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@k8s-master-1 ~]$ for f in k8s-master-2 k8s-master-3; do scp &#x2F;etc&#x2F;keepalived&#x2F;check_apiserver.sh &#x2F;etc&#x2F;keepalived&#x2F;keepalived.conf root@$f:&#x2F;etc&#x2F;keepalived; scp &#x2F;etc&#x2F;haproxy&#x2F;haproxy.cfg root@$f:&#x2F;etc&#x2F;haproxy; done</span><br></pre></td></tr></table></figure>

<p>注意：别忘了更改我们在上面针对k8s-master-2和3讨论的<code>keepalived.conf</code>文件中的两个参数。</p>
<p>如果防火墙在主节点上运行，则在所有三个主节点上添加以下防火墙规则</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ sudo firewall-cmd --add-rich-rule&#x3D;&#39;rule protocol value&#x3D;&quot;vrrp&quot; accept&#39; --permanent</span><br><span class="line">$ sudo firewall-cmd --permanent --add-port&#x3D;8443&#x2F;tcp</span><br><span class="line">$ sudo firewall-cmd --reload</span><br></pre></td></tr></table></figure>

<p>现在，使用以下命令在所有三个主节点上启动并启用keepalived和haproxy服务：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ sudo systemctl enable keepalived --now</span><br><span class="line">$ sudo systemctl enable haproxy --now</span><br></pre></td></tr></table></figure>

<p>这些服务成功启动后，请验证是否已在k8s-master-1节点上启用了VIP（虚拟IP），因为我们已在keepalived配置文件中将k8s-master-1标记为MASTER节点。</p>
<p><img src="http://iubest.gitee.io/pic/640-1601017115394.png" alt="img"></p>
<p>完美，上面的输出确认已在k8s-master-1上启用了VIP。</p>
<h4 id="步骤3-禁用Swap分区，将SELinux设置为主节点和辅助节点的许可规则和防火墙规则"><a href="#步骤3-禁用Swap分区，将SELinux设置为主节点和辅助节点的许可规则和防火墙规则" class="headerlink" title="步骤3. 禁用Swap分区，将SELinux设置为主节点和辅助节点的许可规则和防火墙规则"></a>步骤3. 禁用Swap分区，将SELinux设置为主节点和辅助节点的许可规则和防火墙规则</h4><p>在所有节点（包括工作节点）上禁用交换空间，运行以下命令：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ sudo swapoff -a </span><br><span class="line">$ sudo sed -i &#39;&#x2F; swap &#x2F; s&#x2F;^\(.*\)$&#x2F;#\1&#x2F;g&#39; &#x2F;etc&#x2F;fstab</span><br></pre></td></tr></table></figure>

<p>在所有主节点和工作节点上将SELinux设置为Permissive，运行以下命令，</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">$ sudo setenforce 0</span><br><span class="line">$ sudo sed -i&#39;s &#x2F; ^ SELINUX &#x3D; enforcing $ &#x2F; SELINUX &#x3D; permissive &#x2F;&#39;&#x2F; etc &#x2F; selinux &#x2F; config</span><br></pre></td></tr></table></figure>

<p>主节点的防火墙规则：</p>
<p>如果防火墙在主节点上运行，则在防火墙中允许以下端口</p>
<p><img src="http://iubest.gitee.io/pic/640-1601017115113.webp" alt="img"></p>
<p>所有主节点上运行以下firewall-cmd命令:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">$ sudo firewall-cmd --permanent --add-port&#x3D;6443&#x2F;tcp</span><br><span class="line">$ sudo firewall-cmd --permanent --add-port&#x3D;2379-2380&#x2F;tcp</span><br><span class="line">$ sudo firewall-cmd --permanent --add-port&#x3D;10250&#x2F;tcp</span><br><span class="line">$ sudo firewall-cmd --permanent --add-port&#x3D;10251&#x2F;tcp</span><br><span class="line">$ sudo firewall-cmd --permanent --add-port&#x3D;10252&#x2F;tcp</span><br><span class="line">$ sudo firewall-cmd --permanent --add-port&#x3D;179&#x2F;tcp</span><br><span class="line">$ sudo firewall-cmd --permanent --add-port&#x3D;4789&#x2F;udp</span><br><span class="line">$ sudo firewall-cmd --reload</span><br><span class="line">$ sudo modprobe br_netfilter</span><br><span class="line">$ sudo sh -c &quot;echo &#39;1&#39; &gt; &#x2F;proc&#x2F;sys&#x2F;net&#x2F;bridge&#x2F;bridge-nf-call-iptables&quot;</span><br><span class="line">$ sudo sh -c &quot;echo &#39;1&#39; &gt; &#x2F;proc&#x2F;sys&#x2F;net&#x2F;ipv4&#x2F;ip_forward&quot;</span><br></pre></td></tr></table></figure>

<p>工作节点的防火墙规则：</p>
<p>如果防火墙在工作节点上运行，则在所有工作程序节点上允许防火墙中的以下端口</p>
<p><img src="http://iubest.gitee.io/pic/640-1601017115109.webp" alt="img"></p>
<p>在所有工作节点上运行以下命令:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">$ sudo firewall-cmd --permanent --add-port&#x3D;10250&#x2F;tcp</span><br><span class="line">$ sudo firewall-cmd --permanent --add-port&#x3D;30000-32767&#x2F;tcp                                                   </span><br><span class="line">$ sudo firewall-cmd --permanent --add-port&#x3D;179&#x2F;tcp</span><br><span class="line">$ sudo firewall-cmd --permanent --add-port&#x3D;4789&#x2F;udp</span><br><span class="line">$ sudo firewall-cmd --reload</span><br><span class="line">$ sudo modprobe br_netfilter</span><br><span class="line">$ sudo sh -c &quot;echo &#39;1&#39; &gt; &#x2F;proc&#x2F;sys&#x2F;net&#x2F;bridge&#x2F;bridge-nf-call-iptables&quot;</span><br><span class="line">$ sudo sh -c &quot;echo &#39;1&#39; &gt; &#x2F;proc&#x2F;sys&#x2F;net&#x2F;ipv4&#x2F;ip_forward&quot;</span><br></pre></td></tr></table></figure>

<h4 id="步骤4-在主节点和工作节点上安装容器运行（CRI）Docker"><a href="#步骤4-在主节点和工作节点上安装容器运行（CRI）Docker" class="headerlink" title="步骤4. 在主节点和工作节点上安装容器运行（CRI）Docker"></a>步骤4. 在主节点和工作节点上安装容器运行（CRI）Docker</h4><p>在所有主节点和工作节点上安装Docker，运行以下命令</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ sudo yum install -y yum-utils</span><br><span class="line">$ sudo yum-config-manager --add-repo https:&#x2F;&#x2F;download.docker.com&#x2F;linux&#x2F;centos&#x2F;docker-ce.repo</span><br><span class="line">$ sudo yum install docker-ce -y</span><br></pre></td></tr></table></figure>

<p>运行以下systemctl命令以启动并启用docker服务（在所有主节点和工作节点上也运行此命令）</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ sudo systemctl enable docker --now</span><br></pre></td></tr></table></figure>

<p>现在，让我们在下一步中安装kubeadm，kubelet和kubectl</p>
<h4 id="步骤5-安装Kubeadm，kubelet和kubectl"><a href="#步骤5-安装Kubeadm，kubelet和kubectl" class="headerlink" title="步骤5. 安装Kubeadm，kubelet和kubectl"></a>步骤5. 安装Kubeadm，kubelet和kubectl</h4><p>在所有主节点和辅助节点上安装kubeadm，kubelet和kubectl。首先安装这些软件包之前，我们必须配置Kubernetes repo仓库，在每个主节点和工作节点上运行以下命令</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line">cat &lt;&lt;EOF | sudo tee &#x2F;etc&#x2F;yum.repos.d&#x2F;kubernetes.repo</span><br><span class="line">[kubernetes]</span><br><span class="line">name&#x3D;Kubernetes</span><br><span class="line">baseurl&#x3D;https:&#x2F;&#x2F;packages.cloud.google.com&#x2F;yum&#x2F;repos&#x2F;kubernetes-el7-\$basearch</span><br><span class="line">enabled&#x3D;1</span><br><span class="line">gpgcheck&#x3D;1</span><br><span class="line">repo_gpgcheck&#x3D;1</span><br><span class="line">gpgkey&#x3D;https:&#x2F;&#x2F;packages.cloud.google.com&#x2F;yum&#x2F;doc&#x2F;yum-key.gpg https:&#x2F;&#x2F;packages.cloud.google.com&#x2F;yum&#x2F;doc&#x2F;rpm-package-key.gpg</span><br><span class="line">exclude&#x3D;kubelet kubeadm kubectl</span><br><span class="line">EOF</span><br></pre></td></tr></table></figure>

<p>现在，在yum命令下面运行以安装这些软件包</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ sudo yum install -y kubelet kubeadm kubectl --disableexcludes&#x3D;kubernetes</span><br></pre></td></tr></table></figure>

<p>运行以下systemctl命令以在所有节点（主节点和工作节点）上启用kubelet服务</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">$ sudo systemctl enable kubelet --now</span><br></pre></td></tr></table></figure>

<h4 id="步骤6-从第一个主节点初始化Kubernetes集群"><a href="#步骤6-从第一个主节点初始化Kubernetes集群" class="headerlink" title="步骤6. 从第一个主节点初始化Kubernetes集群"></a>步骤6. 从第一个主节点初始化Kubernetes集群</h4><p>现在到第一个主节点终端下执行以下命令</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@k8s-master-1 ~]$ sudo kubeadm init --control-plane-endpoint &quot;vip-k8s-master:8443&quot; --upload-certs</span><br></pre></td></tr></table></figure>

<p>在以上命令中， <code>--control-plane-endpoint</code>设置了负载均衡器（kube-apiserver）的dns名称和端口，在我的情况下，dns名称是“vip-k8s-master”，端口是“ 8443”，除了此<code>--upload-certs</code>的选项将自动在主节点之间共享证书。</p>
<p>kubeadm命令的输出如下所示：</p>
<p><img src="http://iubest.gitee.io/pic/640-1601017115320.png" alt="img"></p>
<p>很好，上面的输出确认Kubernetes集群已成功初始化。在输出中，我们还获得了其他主节点和工作节点加入集群的命令。</p>
<p>注意：建议将此输出复制到文本文件中，以备将来参考。</p>
<p>运行以下命令，以允许本地用户使用kubectl命令与集群进行交互</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@k8s-master-1 ~]$ mkdir -p $HOME&#x2F;.kube</span><br><span class="line">[kadmin@k8s-master-1 ~]$ sudo cp -i &#x2F;etc&#x2F;kubernetes&#x2F;admin.conf $HOME&#x2F;.kube&#x2F;config</span><br><span class="line">[kadmin@k8s-master-1 ~]$ sudo chown $(id -u):$(id -g) $HOME&#x2F;.kube&#x2F;config</span><br><span class="line">[kadmin@k8s-master-1 ~]$</span><br></pre></td></tr></table></figure>

<p>现在，让我们部署Pod网络（CNI –容器网络接口），在我的情况下，我将calico插件部署为Pod网络，请按照kubectl命令运行</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@k8s-master-1 ~]$ kubectl apply -f https:&#x2F;&#x2F;docs.projectcalico.org&#x2F;v3.14&#x2F;manifests&#x2F;calico.yaml</span><br></pre></td></tr></table></figure>

<p>成功部署Pod网络后，将其余两个主节点添加到群集中。只需从输出中复制用于主节点的命令以加入集群，然后将其粘贴到k8s-master-2和k8s-master-3上，示例如下所示</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@k8s-master-2 ~]$ sudo kubeadm join vip-k8s-master:8443 --token tun848.2hlz8uo37jgy5zqt  --discovery-token-ca-cert-hash sha256:d035f143d4bea38d54a3d827729954ab4b1d9620631ee330b8f3fbc70324abc5 --control-plane --certificate-key a0b31bb346e8d819558f8204d940782e497892ec9d3d74f08d1c0376dc3d3ef4</span><br></pre></td></tr></table></figure>

<p>输出如下：</p>
<p><img src="http://iubest.gitee.io/pic/640-1601017115259.png" alt="img"></p>
<p>以上输出确认k8s-master-3也已成功加入集群。让我们从kubectl命令验证节点状态，到master-1节点并执行以下命令</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@k8s-master-1 ~]$ kubectl get nodes</span><br><span class="line">NAME           STATUS   ROLES    AGE     VERSION</span><br><span class="line">k8s-master-1   Ready    master   31m     v1.18.6</span><br><span class="line">k8s-master-2   Ready    master   10m     v1.18.6</span><br><span class="line">k8s-master-3   Ready    master   3m47s   v1.18.6</span><br><span class="line">[kadmin@k8s-master-1 ~]$</span><br></pre></td></tr></table></figure>

<p>完美，我们三个主节点均已准备就绪，并加入了集群。</p>
<h4 id="步骤7-将Worker节点加入Kubernetes集群"><a href="#步骤7-将Worker节点加入Kubernetes集群" class="headerlink" title="步骤7. 将Worker节点加入Kubernetes集群"></a>步骤7. 将Worker节点加入Kubernetes集群</h4><p>要将工作节点加入集群，请从输出中复制工作程序节点的命令，并将其粘贴到两个工作程序节点上，示例如下所示：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@k8s-worker-1 ~]$ sudo kubeadm join vip-k8s-master:8443 --token tun848.2hlz8uo37jgy5zqt --discovery-token-ca-cert-hash sha256:d035f143d4bea38d54a3d827729954ab4b1d9620631ee330b8f3fbc70324abc5</span><br><span class="line"></span><br><span class="line">[kadmin@k8s-worker-2 ~]$ sudo kubeadm join vip-k8s-master:8443 --token tun848.2hlz8uo37jgy5zqt --discovery-token-ca-cert-hash sha256:d035f143d4bea38d54a3d827729954ab4b1d9620631ee330b8f3fbc70324abc5</span><br></pre></td></tr></table></figure>

<p>输出如下所示：</p>
<p><img src="http://iubest.gitee.io/pic/640-1601017115393.png" alt="img"></p>
<p>现在转到k8s-master-1节点并在kubectl命令下运行以获取状态工作程序节点：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@k8s-master-1 ~]$ kubectl get nodes</span><br><span class="line">NAME           STATUS   ROLES    AGE     VERSION</span><br><span class="line">k8s-master-1   Ready    master   43m     v1.18.6</span><br><span class="line">k8s-master-2   Ready    master   21m     v1.18.6</span><br><span class="line">k8s-master-3   Ready    master   15m     v1.18.6</span><br><span class="line">k8s-worker-1   Ready    &lt;none&gt;   6m11s   v1.18.6</span><br><span class="line">k8s-worker-2   Ready    &lt;none&gt;   5m22s   v1.18.6</span><br><span class="line">[kadmin@k8s-master-1 ~]$</span><br></pre></td></tr></table></figure>

<p>以上输出确认两个工作节点都已加入集群并处于就绪状态。</p>
<p>运行以下命令，以验证在kube-system名称空间中部署的状态。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@k8s-master-1 ~]$ kubectl get pods -n kube-system</span><br></pre></td></tr></table></figure>

<p><img src="http://iubest.gitee.io/pic/640-1601017115315.webp" alt="img"></p>
<h4 id="步骤8-测试Kubernetes集群的高可用性"><a href="#步骤8-测试Kubernetes集群的高可用性" class="headerlink" title="步骤8. 测试Kubernetes集群的高可用性"></a>步骤8. 测试Kubernetes集群的高可用性</h4><p>让我们尝试使用负载平衡器dns名称和端口从远程计算机（CentOS系统）连接到群集。首先，在远程计算机上，我们必须安装kubectl软件包。运行以下命令来设置kubernetes软件仓库。</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">cat &lt;&lt;EOF | sudo tee &#x2F;etc&#x2F;yum.repos.d&#x2F;kubernetes.repo</span><br><span class="line">[kubernetes]</span><br><span class="line">name&#x3D;Kubernetes</span><br><span class="line">baseurl&#x3D;https:&#x2F;&#x2F;packages.cloud.google.com&#x2F;yum&#x2F;repos&#x2F;kubernetes-el7-\$basearch</span><br><span class="line">enabled&#x3D;1</span><br><span class="line">gpgcheck&#x3D;1</span><br><span class="line">repo_gpgcheck&#x3D;1</span><br><span class="line">gpgkey&#x3D;https:&#x2F;&#x2F;packages.cloud.google.com&#x2F;yum&#x2F;doc&#x2F;yum-key.gpg https:&#x2F;&#x2F;packages.cloud.google.com&#x2F;yum&#x2F;doc&#x2F;rpm-package-key.gpg</span><br><span class="line">exclude&#x3D;kubelet kubeadm kubectl</span><br><span class="line">EOF</span><br><span class="line"></span><br><span class="line">$ sudo yum install -y  kubectl --disableexcludes&#x3D;kubernetes</span><br></pre></td></tr></table></figure>

<p>现在在<code>/etc/host</code>文件中添加以下条目:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">192.168.1.45   vip-k8s-master</span><br></pre></td></tr></table></figure>

<p>创建kube目录，并将<code>/etc/kubernetes/admin.conf</code>文件从k8s-master-1节点复制到<code>$HOME/.kube/config</code></p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">$ mkdir -p $HOME&#x2F;.kube</span><br><span class="line">$ scp root@192.168.1.40:&#x2F;etc&#x2F;kubernetes&#x2F;admin.conf $HOME&#x2F;.kube&#x2F;config</span><br><span class="line">$ sudo chown $(id -u):$(id -g) $HOME&#x2F;.kube&#x2F;config</span><br></pre></td></tr></table></figure>

<p>现在运行“kubectl get nodes”命令，</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@localhost ~]$ kubectl get nodes</span><br><span class="line">NAME           STATUS   ROLES    AGE    VERSION</span><br><span class="line">k8s-master-1   Ready    master   3h5m   v1.18.6</span><br><span class="line">k8s-master-2   Ready    master   163m   v1.18.6</span><br><span class="line">k8s-master-3   Ready    master   157m   v1.18.6</span><br><span class="line">k8s-worker-1   Ready    &lt;none&gt;   148m   v1.18.6</span><br><span class="line">k8s-worker-2   Ready    &lt;none&gt;   147m   v1.18.6</span><br><span class="line">[kadmin@localhost ~]$</span><br></pre></td></tr></table></figure>

<p>让我们创建一个名为nginx-lab的部署，镜像为“ nginx ”，然后将该部署作为类型为“ NodePort ”的服务公开</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@localhost ~]$ kubectl create deployment nginx-lab --image&#x3D;nginx</span><br><span class="line">deployment.apps&#x2F;nginx-lab created</span><br><span class="line">[kadmin@localhost ~]$</span><br><span class="line">[kadmin@localhost ~]$ kubectl get deployments.apps nginx-lab</span><br><span class="line">NAME        READY   UP-TO-DATE   AVAILABLE   AGE</span><br><span class="line">nginx-lab   1&#x2F;1     1            1           59s</span><br><span class="line">[kadmin@localhost ~]$ kubectl get pods</span><br><span class="line">NAME                         READY   STATUS    RESTARTS   AGE</span><br><span class="line">nginx-lab-5df4577d49-rzv9q   1&#x2F;1     Running   0          68s</span><br><span class="line">test-844b65666c-pxpkh        1&#x2F;1     Running   3          154m</span><br><span class="line">[kadmin@localhost ~]$</span><br></pre></td></tr></table></figure>

<p>让我们尝试将副本从1扩到4，运行以下命令:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@localhost ~]$ kubectl scale deployment nginx-lab --replicas&#x3D;4</span><br><span class="line">deployment.apps&#x2F;nginx-lab scaled</span><br><span class="line">[kadmin@localhost ~]$</span><br><span class="line">[kadmin@localhost ~]$ kubectl get deployments.apps nginx-lab</span><br><span class="line">NAME        READY   UP-TO-DATE   AVAILABLE   AGE</span><br><span class="line">nginx-lab   4&#x2F;4     4            4           3m10s</span><br><span class="line">[kadmin@localhost ~]$</span><br></pre></td></tr></table></figure>

<p>现在将部署作为服务公开，运行以下命令:</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@localhost ~]$ kubectl expose deployment nginx-lab --name&#x3D;nginx-lab --type&#x3D;NodePort --port&#x3D;80 --target-port&#x3D;80</span><br><span class="line">service&#x2F;nginx-lab exposed</span><br><span class="line">[kadmin@localhost ~]$</span><br></pre></td></tr></table></figure>

<p>获取端口详细信息，并尝试使用curl访问nginx Web服务器</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@localhost ~]$ kubectl get svc nginx-lab</span><br><span class="line">NAME        TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE</span><br><span class="line">nginx-lab   NodePort   10.102.32.29   &lt;none&gt;        80:31766&#x2F;TCP   60s</span><br><span class="line">[kadmin@localhost ~]$</span><br></pre></td></tr></table></figure>

<p>要访问nginx Web服务器，我们可以使用任何主节点或工作节点IP和端口作为“ 31766”</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[kadmin@localhost ~]$ curl http:&#x2F;&#x2F;192.168.1.44:31766</span><br></pre></td></tr></table></figure>

<p>输出如下所示：</p>
<p><img src="http://iubest.gitee.io/pic/640-1601017115391.png" alt="img"></p>
<p>完美，这严重了我们已经在CentOS 7服务器上使用kubeadm成功部署了高可用性Kubernetes集群。</p>
 
      <!-- reward -->
      
      <div id="reword-out">
        <div id="reward-btn">
          打赏
        </div>
      </div>
      
    </div>
    

    <!-- copyright -->
    
    <div class="declare">
      <ul class="post-copyright">
        <li>
          <i class="ri-copyright-line"></i>
          <strong>版权声明： </strong>
          
          本博客所有文章除特别声明外，著作权归作者所有。转载请注明出处！
          
        </li>
      </ul>
    </div>
    
    <footer class="article-footer">
       
<div class="share-btn">
      <span class="share-sns share-outer">
        <i class="ri-share-forward-line"></i>
        分享
      </span>
      <div class="share-wrap">
        <i class="arrow"></i>
        <div class="share-icons">
          
          <a class="weibo share-sns" href="javascript:;" data-type="weibo">
            <i class="ri-weibo-fill"></i>
          </a>
          <a class="weixin share-sns wxFab" href="javascript:;" data-type="weixin">
            <i class="ri-wechat-fill"></i>
          </a>
          <a class="qq share-sns" href="javascript:;" data-type="qq">
            <i class="ri-qq-fill"></i>
          </a>
          <a class="douban share-sns" href="javascript:;" data-type="douban">
            <i class="ri-douban-line"></i>
          </a>
          <!-- <a class="qzone share-sns" href="javascript:;" data-type="qzone">
            <i class="icon icon-qzone"></i>
          </a> -->
          
          <a class="facebook share-sns" href="javascript:;" data-type="facebook">
            <i class="ri-facebook-circle-fill"></i>
          </a>
          <a class="twitter share-sns" href="javascript:;" data-type="twitter">
            <i class="ri-twitter-fill"></i>
          </a>
          <a class="google share-sns" href="javascript:;" data-type="google">
            <i class="ri-google-fill"></i>
          </a>
        </div>
      </div>
</div>

<div class="wx-share-modal">
    <a class="modal-close" href="javascript:;"><i class="ri-close-circle-line"></i></a>
    <p>扫一扫，分享到微信</p>
    <div class="wx-qrcode">
      <img src="//api.qrserver.com/v1/create-qr-code/?size=150x150&data=http://example.com/2020/11/11/k8s/%E4%BD%BF%E7%94%A8Kubeadm%E8%AE%BE%E7%BD%AEKubernetes%E9%9B%86%E7%BE%A4%E7%9A%84%E9%AB%98%E5%8F%AF%E7%94%A8%E6%80%A7/" alt="微信分享二维码">
    </div>
</div>

<div id="share-mask"></div>  
  <ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/k8s/" rel="tag">k8s</a></li></ul>

    </footer>
  </div>

   
  <nav class="article-nav">
    
      <a href="/2020/11/11/k8s/kubernetes%20%E5%B8%B8%E8%A7%81%E9%97%AE%E9%A2%98%E6%80%BB%E7%BB%93/" class="article-nav-link">
        <strong class="article-nav-caption">上一篇</strong>
        <div class="article-nav-title">
          
            kubernetes 常见问题总结.md
          
        </div>
      </a>
    
    
      <a href="/2020/11/11/k8s/%E5%A6%82%E4%BD%95%E7%94%A8%20K8S%20%E6%90%9E%E5%AE%9A%201000%20%E4%B8%AA%E5%BA%94%E7%94%A8%E7%9A%84%E6%B5%8B%E8%AF%95%E7%8E%AF%E5%A2%83/" class="article-nav-link">
        <strong class="article-nav-caption">下一篇</strong>
        <div class="article-nav-title">如何用 K8S 搞定 1000 个应用的测试环境.md</div>
      </a>
    
  </nav>

   
<!-- valine评论 -->
<div id="vcomments-box">
  <div id="vcomments"></div>
</div>
<script src="//cdn1.lncld.net/static/js/3.0.4/av-min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/valine@1.4.14/dist/Valine.min.js"></script>
<script>
  new Valine({
    el: "#vcomments",
    app_id: "",
    app_key: "",
    path: window.location.pathname,
    avatar: "monsterid",
    placeholder: "给我的文章加点评论吧~",
    recordIP: true,
  });
  const infoEle = document.querySelector("#vcomments .info");
  if (infoEle && infoEle.childNodes && infoEle.childNodes.length > 0) {
    infoEle.childNodes.forEach(function (item) {
      item.parentNode.removeChild(item);
    });
  }
</script>
<style>
  #vcomments-box {
    padding: 5px 30px;
  }

  @media screen and (max-width: 800px) {
    #vcomments-box {
      padding: 5px 0px;
    }
  }

  #vcomments-box #vcomments {
    background-color: #fff;
  }

  .v .vlist .vcard .vh {
    padding-right: 20px;
  }

  .v .vlist .vcard {
    padding-left: 10px;
  }
</style>

 
     
</article>

</section>
      <footer class="footer">
  <div class="outer">
    <ul>
      <li>
        Copyrights &copy;
        2015-2020
        <i class="ri-heart-fill heart_icon"></i> TzWind
      </li>
    </ul>
    <ul>
      <li>
        
        
        
        由 <a href="https://hexo.io" target="_blank">Hexo</a> 强力驱动
        <span class="division">|</span>
        主题 - <a href="https://github.com/Shen-Yu/hexo-theme-ayer" target="_blank">Ayer</a>
        
      </li>
    </ul>
    <ul>
      <li>
        
        
        <span>
  <span><i class="ri-user-3-fill"></i>访问人数:<span id="busuanzi_value_site_uv"></span></s>
  <span class="division">|</span>
  <span><i class="ri-eye-fill"></i>浏览次数:<span id="busuanzi_value_page_pv"></span></span>
</span>
        
      </li>
    </ul>
    <ul>
      
    </ul>
    <ul>
      
    </ul>
    <ul>
      <li>
        <!-- cnzz统计 -->
        
        <script type="text/javascript" src='https://s9.cnzz.com/z_stat.php?id=1278069914&amp;web_id=1278069914'></script>
        
      </li>
    </ul>
  </div>
</footer>
      <div class="float_btns">
        <div class="totop" id="totop">
  <i class="ri-arrow-up-line"></i>
</div>

<div class="todark" id="todark">
  <i class="ri-moon-line"></i>
</div>

      </div>
    </main>
    <aside class="sidebar on">
      <button class="navbar-toggle"></button>
<nav class="navbar">
  
  <div class="logo">
    <a href="/"><img src="/images/ayer-side.svg" alt="Hexo"></a>
  </div>
  
  <ul class="nav nav-main">
    
    <li class="nav-item">
      <a class="nav-item-link" href="/">主页</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/archives">归档</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/categories">分类</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/tags">标签</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" target="_blank" rel="noopener" href="http://www.baidu.com">百度</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/friends">友链</a>
    </li>
    
    <li class="nav-item">
      <a class="nav-item-link" href="/2019/about">关于我</a>
    </li>
    
  </ul>
</nav>
<nav class="navbar navbar-bottom">
  <ul class="nav">
    <li class="nav-item">
      
      <a class="nav-item-link nav-item-search"  title="搜索">
        <i class="ri-search-line"></i>
      </a>
      
      
      <a class="nav-item-link" target="_blank" href="/atom.xml" title="RSS Feed">
        <i class="ri-rss-line"></i>
      </a>
      
    </li>
  </ul>
</nav>
<div class="search-form-wrap">
  <div class="local-search local-search-plugin">
  <input type="search" id="local-search-input" class="local-search-input" placeholder="Search...">
  <div id="local-search-result" class="local-search-result"></div>
</div>
</div>
    </aside>
    <script>
      if (window.matchMedia("(max-width: 768px)").matches) {
        document.querySelector('.content').classList.remove('on');
        document.querySelector('.sidebar').classList.remove('on');
      }
    </script>
    <div id="mask"></div>

<!-- #reward -->
<div id="reward">
  <span class="close"><i class="ri-close-line"></i></span>
  <p class="reward-p"><i class="ri-cup-line"></i>请我喝杯咖啡吧~</p>
  <div class="reward-box">
    
    
  </div>
</div>
    
<script src="/js/jquery-2.0.3.min.js"></script>


<script src="/js/lazyload.min.js"></script>

<!-- Tocbot -->


<script src="/js/tocbot.min.js"></script>

<script>
  tocbot.init({
    tocSelector: '.tocbot',
    contentSelector: '.article-entry',
    headingSelector: 'h1, h2, h3, h4, h5, h6',
    hasInnerContainers: true,
    scrollSmooth: true,
    scrollContainer: 'main',
    positionFixedSelector: '.tocbot',
    positionFixedClass: 'is-position-fixed',
    fixedSidebarOffset: 'auto'
  });
</script>

<script src="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.js"></script>
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/jquery-modal@0.9.2/jquery.modal.min.css">
<script src="https://cdn.jsdelivr.net/npm/justifiedGallery@3.7.0/dist/js/jquery.justifiedGallery.min.js"></script>

<script src="/dist/main.js"></script>

<!-- ImageViewer -->

<!-- Root element of PhotoSwipe. Must have class pswp. -->
<div class="pswp" tabindex="-1" role="dialog" aria-hidden="true">

    <!-- Background of PhotoSwipe. 
         It's a separate element as animating opacity is faster than rgba(). -->
    <div class="pswp__bg"></div>

    <!-- Slides wrapper with overflow:hidden. -->
    <div class="pswp__scroll-wrap">

        <!-- Container that holds slides. 
            PhotoSwipe keeps only 3 of them in the DOM to save memory.
            Don't modify these 3 pswp__item elements, data is added later on. -->
        <div class="pswp__container">
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
            <div class="pswp__item"></div>
        </div>

        <!-- Default (PhotoSwipeUI_Default) interface on top of sliding area. Can be changed. -->
        <div class="pswp__ui pswp__ui--hidden">

            <div class="pswp__top-bar">

                <!--  Controls are self-explanatory. Order can be changed. -->

                <div class="pswp__counter"></div>

                <button class="pswp__button pswp__button--close" title="Close (Esc)"></button>

                <button class="pswp__button pswp__button--share" style="display:none" title="Share"></button>

                <button class="pswp__button pswp__button--fs" title="Toggle fullscreen"></button>

                <button class="pswp__button pswp__button--zoom" title="Zoom in/out"></button>

                <!-- Preloader demo http://codepen.io/dimsemenov/pen/yyBWoR -->
                <!-- element will get class pswp__preloader--active when preloader is running -->
                <div class="pswp__preloader">
                    <div class="pswp__preloader__icn">
                        <div class="pswp__preloader__cut">
                            <div class="pswp__preloader__donut"></div>
                        </div>
                    </div>
                </div>
            </div>

            <div class="pswp__share-modal pswp__share-modal--hidden pswp__single-tap">
                <div class="pswp__share-tooltip"></div>
            </div>

            <button class="pswp__button pswp__button--arrow--left" title="Previous (arrow left)">
            </button>

            <button class="pswp__button pswp__button--arrow--right" title="Next (arrow right)">
            </button>

            <div class="pswp__caption">
                <div class="pswp__caption__center"></div>
            </div>

        </div>

    </div>

</div>

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.css">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/default-skin/default-skin.min.css">
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/photoswipe@4.1.3/dist/photoswipe-ui-default.min.js"></script>

<script>
    function viewer_init() {
        let pswpElement = document.querySelectorAll('.pswp')[0];
        let $imgArr = document.querySelectorAll(('.article-entry img:not(.reward-img)'))

        $imgArr.forEach(($em, i) => {
            $em.onclick = () => {
                // slider展开状态
                // todo: 这样不好，后面改成状态
                if (document.querySelector('.left-col.show')) return
                let items = []
                $imgArr.forEach(($em2, i2) => {
                    let img = $em2.getAttribute('data-idx', i2)
                    let src = $em2.getAttribute('data-target') || $em2.getAttribute('src')
                    let title = $em2.getAttribute('alt')
                    // 获得原图尺寸
                    const image = new Image()
                    image.src = src
                    items.push({
                        src: src,
                        w: image.width || $em2.width,
                        h: image.height || $em2.height,
                        title: title
                    })
                })
                var gallery = new PhotoSwipe(pswpElement, PhotoSwipeUI_Default, items, {
                    index: parseInt(i)
                });
                gallery.init()
            }
        })
    }
    viewer_init()
</script>

<!-- MathJax -->

<!-- Katex -->

<!-- busuanzi  -->


<script src="/js/busuanzi-2.3.pure.min.js"></script>


<!-- ClickLove -->

<!-- ClickBoom1 -->

<!-- ClickBoom2 -->

<!-- CodeCopy -->


<link rel="stylesheet" href="/css/clipboard.css">

<script src="https://cdn.jsdelivr.net/npm/clipboard@2/dist/clipboard.min.js"></script>
<script>
  function wait(callback, seconds) {
    var timelag = null;
    timelag = window.setTimeout(callback, seconds);
  }
  !function (e, t, a) {
    var initCopyCode = function(){
      var copyHtml = '';
      copyHtml += '<button class="btn-copy" data-clipboard-snippet="">';
      copyHtml += '<i class="ri-file-copy-2-line"></i><span>COPY</span>';
      copyHtml += '</button>';
      $(".highlight .code pre").before(copyHtml);
      $(".article pre code").before(copyHtml);
      var clipboard = new ClipboardJS('.btn-copy', {
        target: function(trigger) {
          return trigger.nextElementSibling;
        }
      });
      clipboard.on('success', function(e) {
        let $btn = $(e.trigger);
        $btn.addClass('copied');
        let $icon = $($btn.find('i'));
        $icon.removeClass('ri-file-copy-2-line');
        $icon.addClass('ri-checkbox-circle-line');
        let $span = $($btn.find('span'));
        $span[0].innerText = 'COPIED';
        
        wait(function () { // 等待两秒钟后恢复
          $icon.removeClass('ri-checkbox-circle-line');
          $icon.addClass('ri-file-copy-2-line');
          $span[0].innerText = 'COPY';
        }, 2000);
      });
      clipboard.on('error', function(e) {
        e.clearSelection();
        let $btn = $(e.trigger);
        $btn.addClass('copy-failed');
        let $icon = $($btn.find('i'));
        $icon.removeClass('ri-file-copy-2-line');
        $icon.addClass('ri-time-line');
        let $span = $($btn.find('span'));
        $span[0].innerText = 'COPY FAILED';
        
        wait(function () { // 等待两秒钟后恢复
          $icon.removeClass('ri-time-line');
          $icon.addClass('ri-file-copy-2-line');
          $span[0].innerText = 'COPY';
        }, 2000);
      });
    }
    initCopyCode();
  }(window, document);
</script>


<!-- CanvasBackground -->


    
  </div>
</body>

</html>